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1.0  INTRODUCTION 

1.1  Background 

Recent  improvements  in  computer  systems  and  displays  have  enabled  new  simulation  technologies  such  as 
Augmented,  Mixed,  and  Virtual  Environments  (AMVE).  Increased  computer  power  at  low  cost,  wireless 
networks,  miniaturizations  of  sensor  and  computer  components,  and  better  visual,  auditory  and  tactile  display 
systems  are  contributing  to  the  maturation  of  these  technologies.  Potential  applications  in  military  operations, 
as  well  as  training  and  system  design  are  providing  requirements  that  have  spurred  this  technology 
development. 

Today,  most  of  the  attention  is  focused  on  the  development  of  the  technologies  themselves.  However,  to  be 
effective  in  military  operations,  the  technologies  must  evolve  into  systems  that  provide  the  information  that 
their  human  users  need  to  accomplish  military  objectives.  Compared  to  research  on  computer  architectures, 
communication  protocols,  and  display  devices  there  has  been  relatively  little  research  on  the  perceptual 
requirements  for  displays,  human-computer-interaction  issues,  design  of  effective  training  approaches, 
measurement  of  human  perfonnance  and  cultural  and  organizational  issues.  The  fundamental  knowledge 
available  today  already  indicates  a  large  potential  of  AMVE  technology  for  a  broad  spectrum  of  military 
applications. 

An  important  outcome  of  a  previous  workshop  “What  is  essential  for  Virtual  Reality  systems  to  meet  military 
human  performance  goals”  of  the  NATO  Research  Study  Group  HFM-021  in  the  year  2000  was  that: 

•  baseline  applications  of  VR  were  solid  in  the  automotive  industry  and  entertainment  industry,  and 
military  applications  were  beginning  to  emerge  and  be  evaluated, 

•  Successful  application  of  VR  depends  strongly  on 

•  the  quality  of  the  interaction  methods  of  humans  with  VR  (train  like  we  fight), 

•  on  the  on  the  level  of fidelity  of  the  virtual  world,  and 

•  on  the  multidisciplinary  involvement  of  scientists,  engineers,  practitioners  and  users. 

This  workshop  titled  “Virtual  media  for  military  applications”  organized  by  the  NATO  Research  Study  Group 
HFM-136  focused  on  the  broader  set  of  AMVE  technology  for  military  applications.  Military  users  were 
brought  together  with  academic  researchers  and  industry  to  discuss  if  AVME  meets  operational  needs.  It 
provided  a  unique  opportunity  to  assess  advances  in  AMVE  technology  and  to  monitor  the  progress  of  the 
recommended  role  of  human  factors  research  and  user  involvement  in  the  development  of  AMVE 
applications. 
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1.2  Purpose  and  scope  of  this  workshop 

Chairman  Thomas  Alexander  pointed  out  that  the  purpose  of  this  workshop  was: 

•  to  summarize  previous  and  on-going  research  (and  see  if  AMVE  technology  indeed  provides  the 
intuitive  human-system  interaction  expected), 

•  to  identify  the  keys  for  implementation  of  ready  to  use  technology, 

•  to  identify  knowledge  gaps  and  thrusts  and  establish  an  agenda  for  future  efforts  that  explore  the 
human  dimensions  of  virtual  media  for  military  applications. 

The  workshop  concentrated  on  the  following  research  area’s: 

•  Training  methods, 

•  Human  performance  requirements, 

•  Performance  measurement  techniques  and  assessment, 

•  Human  Factors  issues  in  design  and  utilization. 

with  a  focus  on  the  following  application  area’s: 

•  Command,  Control,  Communication,  Computer,  Information,  Surveillance  and  Reconnaissance 
(C4ISR)  systems,  in  particular  user  interface  issues, 

•  Tele-presence,  tele-operation,  and  tele-manipulation  in  reconnaissance,  surveillance,  and  target 
acquisition, 

•  military  training  and  simulation, 

•  mission  preparation  and  rehearsal, 

•  systems  acquisition, 

•  mission  support  (maintenance,  decision  aiding,  logistics,  navigation). 

1.3  Program  workshop 

The  workshop  took  place  over  three  days  with  the  following  structure: 

Tuesday,  June  13,  2006 

•  Keynote  Address  by  professor  Paul  Milgram  (Toronto  University,  Canada):  How  the  concept  of 
Mixed  Reality  Encompasses  Augmented  Reality  and  Virtual  Environments. 

•  Session  1 :  Command  and  Control 

•  Chairs:  Thomas  Alexander  (Research  Establishment  for  Applied  Sciences  FGAN,  Germany)  and 
Patrik  Lif  (Swedish  Defence  Research  Agency  FOI,  Sweden). 

•  Speakers: 

•  Thomas  Alexander  (FGAN,  Germany):  Applicability  of  Virtual  Environments  as  C4ISR. 

•  Jared  Freeman  (Aptima  Inc.  US):  Measuring,  Monitoring,  and  Managing  Knowledge  in 
Command  and  Control  Organizations. 
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•  James  Lussier  (US  Army  Research  Institute,  US):  Components  of  Effective  Learning. 

•  Neville  A.  Stanton  (Brunei  University,  UK):  Experimental  Studies  in  a  Reconfigurable  C4 
Test-bed  for  Network  Enabled  Capability. 

•  Andreas  Tolk  (Virginia  M&S  Center,  US):  Challenges  and  Potential  of  Service-oriented 
Architectures  for  Net-Centric  Operations. 

•  Hendrik-Jan  van  Veen  (TNO  Defense,  Security  and  Safety,  The  Netherlands):  SIMNEC, 
research  platform  for  studying  human  functioning  in  NCW. 

•  Session  2:  Tele-operations 

•  Chairs:  Ebb  Smith  (Defence  Science  and  Technology  Laboratory  DSTL,  UK)  and  Professor 

Robert  Stone  (University  of  Birmingham  and  Director  of  the  Human  Factors  Integration  Defence 

Technology  Centre  (HFI  DTC)). 

•  Speakers: 

•  Robert  J.  Stone  (University  of  Birmingham,  UK):  Serious  Gaming  Technologies  Support 
Human  Factors  Investigations  of  Advanced  Interfaces  For  Semi-autonomous  Vehicles. 

•  Jan  B.F.  van  Erp  (TNO  Defense,  Security  and  Safety,  The  Netherlands):  Tele-presence: 
Bringing  the  Operator  Back  in  the  Loop. 

•  Michael  Barnes  (US  Army  Research  Institute,  US):  Understanding  Soldier  Robot  Teams  in 
Virtual  Environments. 

•  Boris  Trouvain  (FGAN-FKIE,  Germany):  Tele-operation  of  Unmanned  Vehicles,  The 
Human  Factor. 

•  Robert  Taylor  (DSTL,  UK):  Human  Automation  Integration  for  Supervisory  Control  of 
UAVs. 

Wednesday  June  14,  2006 

•  Keynote  Address  by  COL  James  Shufelt  (US  Army,  TRADOC  program  Integration  Office):  The 

Future  Role  of  Virtual  Simulators/Simulations  in  U.S.  Army  Training. 

•  Session  3:  Vehicle  simulation 

•  Chairs:  Lochlan  Magee  (DR&D,  Canada)  and  Sylvain  Hourier  (IMASSA,  France). 

•  Speakers: 

•  LTC  Wil  Riggins  (US  Army,  Program  Executive  Office,  Simulation,  Training  and 
Instrumentation):  Requirements  for  Virtual  Vehicle  Simulation. 

•  Leonhart  Vogelmeier  (EADS,  Military  Aircrafts):  Interaction  Method  for  Virtual  Reality 
Applications. 

•  Brian  Schreiber  (US  Air  Force  Research  Lab  and  Lumir  Research  Institute,  Arizona  ): 
Evaluating  Misiion  Training  Fidelity  requirements,  Examining  key  Issues  in  Deployability 
and  Trainability. 

•  Bemd  de  Graaf  (TNO  Defense,  Security  and  Safety,  The  Netherlands):  Military  Simulation 
Centre:  Vehicle  for  Validation  of  Military  Flight  Simulation. 
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•  Mark  Espanant  (CAE,  Canada):  The  Application  of  Simulation  to  Study  Eiuman  Performance 
Impacts  of  Evolutionary  and  Revolutionary  Changes  to  Armored  Vehicle  Design. 

•  Session  4:  Dismounted  Simulation 

•  Chairs:  Nico  Delleman  (TNO  Defense,  Security  and  Safety,  The  Netherlands)  and  LtCmdr 
Joseph  Cohn  (Naval  Research  Laboratory,  US). 

•  Speakers: 

•  Phillip  Jones  (MYMIC  Low  Liability  Company,  US):  Emerging  Requirements  for 
Dismounted  Virtual  Simulators. 

•  Amela  Sadagic  (MOVES  Institute,  US  Naval  Postgraduate  School):  Combined  Arms 
Training,  Methods  and  Measures  for  a  Changing  World. 

•  Bruce  Knerr  (US  Army  Research  Institute,  US):  Current  Issues  in  the  Use  of  Virtual 
Simulations  fro  Dismounted  Soldier  Training. 

•  Frederick  P.  Brooks,  Jr.  (University  of  North  Carolina,  US):  Challenges  of  Virtual 
Environment  Training  of  Dismounted  Teams. 

•  James  Templeman  (Naval  Research  Lab,  US):  Immersive  Simulation  to  Train  Urban  Infantry 
Combat. 

•  John  Frim  (DR&D,  Canada):Use  of  the  Dismounted  Soldier  Simulator  to  Corroborate  NVG 
Studies  in  a  Field  Setting. 

•  Keynote  Address  by  Bowen  Loftin  (Texas  A&M  University,  US):  The  Future  of  Simulation. 

Thursday  June  15,  2006 

•  Session  5 :  Mixed  and  Augmented  Reality. 

•  Chairs:  Stephen  Goldberg  (US  Army  Research  Institute,  US)  and  Lisbeth  Rasmussen  (Danish 
Defense  Research  Establishment,  Denmark). 

•  Speakers: 

•  Mark  A.  Livingston  (Naval  research  Lab,  US):  Battlefield  Augmented  Reality. 

•  Jarrell  Pair  (University  of  Southern  California,  Institute  for  Simulation  and  Training,  US): 
Performance  Assessment  in  Immersive  Mixed  and  Virtual  Environment  Systems. 

•  Ed  Bachelder  (Systems  technology  Inc.,  US):  Helicopter  Air  Crew  Training  using  Fused  Reality. 

•  Brian  F.  Goldiez  (University  of  central  Florida,  Institute  for  Simulation  and  Training,  US): 
Human  Performance  Assessment  when  using  Augmented  Reality  for  Navigation. 

•  Sheila  Jaszlics  (Pathfinder  Systems  Inc.,  US):  The  DARTS  Augmented  Reality  System. 

•  Matthew  Franklin  (Qinetiq,  UK):  The  Lessons  Learned  in  the  application  of  Augmented  Reality. 

•  Stephen  Ellis  (NASA-ARC,  US):  The  User-interface  to  virtual  or  augmented  environment 
displays  for  the  Air  Traffic  Control. 

•  TER  and  Keynote  Speaker  Comments 

•  prof  Peter  Werkhoven  (TNO  Defense,  Security  and  Safety,  The  Netherlands). 
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•  prof  Paul  Milgram  (Toronto  University,  Canada). 

•  COL  James  Shufelt  (US  Army,  TRADOC  program  Integration  Office). 

•  prof  Bowen  Loftin  (Texas  A&M  University,  US). 


1.4  Attendees 

Distribution  of  attendees  across  nationality  and  affiliation: 


Country 

Total 

Defense 

research 

institutes 

Military 

Industry 

Academia/ 
Civil  res. 
inst. 

Canada 

3 

1 

1 

1 

Denmark 

1 

1 

France 

1 

1 

Germany 

3 

2 

1 

Netherlands 

6 

1 

5 

Sweden 

2 

2 

United  Kingdom 

5 

3 

2 

USA 

20 

9 

3 

3 

5 

Total 

41 

18 

4 

6 

13 

2.0  TECHNICAL-SCIENTIFIC  SITUATION  OF  MILITARY  VR  APPLICATIONS 

2.1  Introduction 

Technological  developments 

•  We  see  a  rise  of  mixed  reality  (Milgram)  technology.  With  current  technology  real  and  virtual  worlds  can 
be  mixed.  The  amount  of  virtual  varies  from  seeing  a  virtual  bird  in  your  real  environment  to  seeing  your 
real  hand  in  a  virtual  environment.  In  the  first  example  only  the  bird  has  to  be  modeled.  In  the  second  the 
complete  environment.  More  complex  examples  include  virtual  in  real  in  virtual.  Milgram  presented  a 
theoretical  framework  to  categorize  various  forms  of  mixed  reality.  From  a  technical  point  of  view  three 
dimensions  can  be  distinguished:  virtual  versus  real,  ego-centricity  versus  exo-centricity  and  display 
congruency.  It  would  be  interesting  to  include  social  dimensions  and  other  modalities  thanvisual. 

•  Industry  claims  that  robust  Augmented  Reality  (AR)  -  representing  virtual  entities  in  real  environments  - 
is  technically  feasible  (Jaszlics).  Furthermore,  the  AR  community  is  supposed  to  be  “big  enough”  to 
increase  Technology  Readiness  level  (TRL)  and  that  robust  AR  is  technically  feasible  (Jaszlics).  Wireless 
stereoscopic  networked  AR  demonstration  systems  are  available  (e.g.  DARTS  system).  However,  some 
technological  challenges  with  respect  to  tracking  and  occlusion  are  still  remaining.  Accurate  tracking  of 
the  body  parts  of  a  soldier  in  the  open  field  is  still  years  away  (Ellis).  But  we  can  start  looking  at 
applications  that  don’t  need  this  accuracy  or  applications  in  instrumented  environments.  The  ability  to 
occlude  parts  of  the  real  world  that  are  covered  by  virtual  elements  generally  still  relies  on  immature 
technology  such  as  real  time  3D  reconstruction  of  the  real  word  and  filtering  the  real  world  (video 
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merging  or  occlusion  displays).  Furthermore  the  resolution  of  (see-through)  head  mounted  display  is 
generally  not  high  enough,  for  example  for  AR  training  of  Forward  Air  Controllers  (Franklin).  Research 
into  the  visual  requirements  for  augmented  reality  displays  for  the  airport  tower  has  shown  that  virtual 
information  that  occludes  the  real  world  may  leave  out  currently  used  dynamic  visual  cues  that  provide 
lead  information  used  for  spacing  and  sequencing  during  air  traffic  management  tasks  (Ellis).  This 
warning  probably  has  a  more  general  scope. 

•  In  controlled  conditions  (e.g.  cockpits  and  helicopter  training)  the  concept  of  fused  reality  may  be  an 
alternative  for  real  time  3D  reconstruction  of  the  real  world.  Fused  reality  makes  use  of  live  video  capture 
and  VR  merged  by  chroma-key  (magenta)  and  luma-key  technology  allowing  real  objects  to  move  in 
virtual  space.  For  example  in  helicopter  training  it  eliminates  the  need  for  expensive  computer  models 
(Bachelder). 

•  The  value  of  3D  sound  in  virtual  training  generally  seems  to  be  underestimated.  For  example,  in  the  case 
of  forward  air  control  3D  sound  can  compensate  for  low  resolution  displays:  we  hear  the  plane  before  we 
see  it  (Franklin). 

•  AR  applications  are  starting  to  look  at  embedded  training  of  team  operations.  The  Battlefield  Augmented 
Reality  System  (BARS)  support  dismounted  soldiers  in  room  clearing  operations.  The  information 
provided  through  head  mounted  displays  changes  on  criticality  and  need.  Voice  and  gesture  commands 
can  be  used  to  interact  with  the  system.  The  research  focus  is  now  on  multiple  users  and  team 
performance.  The  use  of  head- mounted  displays  is  still  cumbersome  and  limits  human  communication. 

•  Training  in  mixed  reality  may  have  specific  effects  on  realism,  presence,  affective  appraisal,  etc.  This 
poses  high  demands  on  cognitive  performance  tests.  Test  batteries  are  needed  to  assess  cognitive 
performance  in  mixed  reality.  A  test  battery  to  assess  attention,  spatial  ability,  memory,  reasoning  abilities 
will  be  completed  in  2007  (pair). 

•  It  has  become  clear  from  many  discussions  on  application  development  that  application  developers  move 
away  from  high-end  high-fidelity  simulators  towards  low-end  simulation.  On  one  hand  head  mounted 
displays  are  still  cumbersome  and  high-end  simulators  are  still  costly.  On  the  other  hand  many  training 
tasks  turn  out  not  to  rely  on  realism  and  can  be  done  using  powerful  commercial  off-the-shelf-game 
engines  (Stone).  It  most  be  noted  however,  that  high-end  simulators  (CAVE,  motion  platforms)  are  still 
required  for  high-fidelity  skill  training  including  eye-hand  coordination  tasks  (e.g.  surgery,  flight 
simulation)  and  multi-user  spatial  awareness  tasks  (e.g.  room  clearing  with  teams).  We  always  need  to  go 
back  to  “what  are  the  training  tasks?”.  High  fidelity  isn’t  always  the  answer.  We  need  to  design  to 
optimize  training  for  specific  tasks  (Brooks). 

•  Interface  technology  has  not  made  much  progress.  The  workshop  has  not  revealed  any  innovations  in  this 
field.  This  is  remarkable  because  a  strong  recommendation  of  a  NATO  workshop  in  2000  discussed  the 
need  to  improve  the  quality  of  interfaces  with  VR.  Tracking  is  still  a  problem,  head-mounted  displays  are 
still  cumbersome,  3D  audio  is  not  used  much,  force  feedback  is  still  in  its  infancy.  Although  tactile 
displays  are  available  for  the  torso  we  haven’t  seen  applications  that  make  use  of  it.  Requirements 
generators  and  hardware  developers  still  express  the  need  for  intuitive  interfaces  allowing  “train  as  you 
fight”  simulations.  In  particular  there  is  a  need  for  improved  ergonomic  design  of  mobile  AR  applications 
for  hands  free  multimodal  interfaces  (Bachelder). 

Scientific  disciplines  involved 

•  Since  the  recognition  of  the  importance  of  intuitive  man-machine  interaction  and  intelligent  system 
adaptation  to  human  conditions,  simulator  engineers  have  involved  psychologists  and  biologists  to 
improve  human  performance  capabilities  within  these  systems. 
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•  Today  we  see  the  need  for  more  disciplines  to  be  included  in  development  of  effective  AMVE 
technologies.  The  research  field  of  simulation  deals  with  modeling  (virtual  and  constructive)  worlds, 
objects  and  behavior,  scenario  generation,  training  methods,  multi-sensory  interfaces  and  didactic 
concepts  but  also  with  drama,  style  and  emotions  (the  gaming  element).  These  components  have  only  had 
limited  application  to  AMVE  applications.  But  this  is  why,  for  example,  screenwriters  now  collaborate 
with  the  U.S.  intelligence  community  on  "war  gaming"  and  why  Paramount  Digital  Entertainment 
collaborates  with  the  US  Department  of  Defense  on  "crisis-management  simulation".  While  it  has  been 
claimed  that  these  components  have  substantial  impact  on  user  experience,  learning  and  training  transfer, 
there  is  only  limited  research  to  backup  the  claims.  So,  the  creative  disciplines  and  liberal  arts  are 
beginning  to  be  involved  in  serious  simulation. 

•  With  the  introduction  of  new  doctrine  and  networked  (joint  and  combined)  command  and  control 
capabilities  (NEC)  new  requirements  for  the  use  of  AMVE  technologies  are  being  developed. 

•  We  need  to  cope  with  a  deluge  of  data  and  information  overload  at  all  levels  (tactical,  operational  and 
strategic)  (Alexander).  How  will  information  be  “tunneled”  to  the  appropriate  user?  How  will 
information  be  filtered  according  to  classification  and  sensitivity  to  ensure  it  gets  to  the  right  people  at 
the  right  time  and  in  the  right  format?  Obviously  other  dimensions  than  technology  become  important 
such  as  culture,  organization,  leadership,  etc. 

•  There  is  a  need  for  a  theoretical  framework  and  predictive  models  on  the  complex  social  and 
organizational  dimensions  as  a  direct  consequence  of  the  potential  of  dynamic  task  allocation, 
common  operational  pictures,  commanders  talking  directly  to  soldiers  and  aircrew,  etc.  There  are  no 
sufficient  theoretical  frameworks  to  model  such  complex  socio-technical  systems.  First  socio- 
technical  systems  that  should  help  leaders  to  measure  monitor  and  manage  are  being  developed 
(Freeman)  but  not  yet  evaluated. 

•  There  is  a  gap  in  knowledge  on  how  quickly  we  can  learn  and  adapt  to  the  use  of  new  technologies. 
How  well  do  people  learn  in  highly  dynamic  structures?  Decisions  may  get  easier  but  actions  may  get 
harder. 

•  Scale  and  complicity  drive  us  towards  concept  development  and  experimentation  environments  with 
an  important  role  of  VR  for  creating  large  scale  distributed  multi-user  virtual  environments  including 
real  and  virtual  humans,  sensors  and  systems  and  complex  decision  structures  and  dependencies. 

•  Conclusion:  New  disciplines  must  urgently  get  involved:  creative  disciplines  (scenario  and  gaming 
elements),  social  disciplines  (dimensions  of  socio-technical  systems),  organizational  disciplines 
(mechanisms  and  mathematics  of  self  organizing  networks). 

Evolved  application  fields 

Command  and  control: 

•  Current  virtual  command  and  control  training  systems  generally  just  provide  monitoring  functionality 
and  lack  sufficient  feedback  and  instructional  mechanisms  (Fussier).  It  is  crucial  to  include  feedback 
mechanisms  and  to  be  able  to  measure  thinking  skills  and  judgments  of  tactical  situations. 

•  Network  enabled  command  and  control  deals  with  dynamic  roles  and  functions,  less  well  defined 
context  for  decision  making,  real  time  information  sharing  and  a  global  scale  of  operations.  Central  in 
network  enabled  command  and  control  is  to  ability  to  share  relevant  information  across  networks  at 
the  right  time  and  in  the  right  form.  Although  technology  currently  enables  common  operational 
pictures  (shared  information),  it  is  still  a  challenge  to  create  common  mental  models  (shared  situation 
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awareness).  Tolk  pointed  out  that  progress  in  this  field  relies  more  on  cultural  factors  (e.g.  the 
willingness  to  share  information,  trusting  others  who  are  at  a  distance  and  allowing  for 
misinterpretations  due  to  cultural  differences)  than  on  technology.  Further,  progress  relies  on  dealing 
with  organizational  issues  rather  than  technological  ones. 

•  Challenges  for  the  use  of  network  enabled  capabilities  (NEC)  are  the  dynamic  reallocation  of 
functions  during  NEC-operations,  and  tools  for  supporting  multiple  command  levels  (van  Veen). 
When  experimenting  with  Networked  Enabled  command  and  control  (C2)  concepts  in  simulated 
worlds  it  must  be  possible  to  vary  these  aspects  and  to  monitor  behavior  and  performance.  The  human 
factors  community  currently  lacks  a  theoretical  framework  and  predictive  models  for  designing  NEC- 
concepts  in  laboratories  or  test  beds.  .  Simulated  worlds  provide  experimentation  environments  for 
exploring  concepts  and  development  processes.  Advanced  simulated  test  beds  facilitate  the  translation 
of  operational  needs  into  new  concepts  and  produce  new  research  questions.  Test  beds  also  provide  a 
means  to  try  out  new  technological  innovations  to  evolve  further  new  concepts  and  capabilities. 

•  Public  sources  such  as  “Google  Earth”  have  become  useful  for  creating  simulated  training 
environments  for  urban  operations  (Stanton). 

Tele-operations: 

•  Fligh  fidelity  virtual  training  environments  for  tele-operations  making  use  of  head-mounted  displays 
have  not  always  brought  the  success  expected.  Stone  opined  that  early  VR  technology  ‘failed  to 
deliver’.  Low-fidelity  gaming  environments  may  provide  a  new  promise.  Stone  used  various 
commercial  game  engines  and  experiments  favoring  a  “quick  and  dirty”  approach  to  human  factors 
research.  This  approach  relies  on  high  frequency  short  cyclic  concept  development  in  which  the 
quality  of  man-machine  interface  is  implicit  and  develops  in  an  evolutionary  manner.  This  seems  to 
be  the  way  the  commercial  gaming  world  works.  Flowever,  it  should  be  noted  that  the  game  industry 
has  been  successful  in  adopting  many  of  the  research  results  that  have  come  from  thorough  academic 
human  factors  studies.  It  is  a  misunderstanding  that  evolutionary  processes  are  sufficient  by 
themselves  and  that  funding  of  n  academic  human  factors  studies  is  a  poor  investment.  For  example, 
within  some  communities  high-fidelity  systems  are  still  thought  to  be  essential  for  effective  training 
transfer  of  perceptual-motor  skills  (flying,  surgery,  etc.)  when  some  recent  research  has  indicated  that 
this  in  not  necessarily  the  case. 

•  Given  our  current  technology,  fully  autonomous  robotic  systems  do  not  seem  to  be  the  right 
approach..  Experiments  (van  Erp)  have  shown  that  human  intelligence  is  still  needed  and  that  the 
operator  must  be  kept  in  the  loop  facilitating  some  sort  of  tele-presence.  The  use  of  HMDs  has  not 
been  shown  to  be  successful  for  tele -presence.  Attention  has  shifted  towards  BOOM-displays.  It 
should  be  noted  that  many  experimental  results  cannot  be  generalized  but  only  hold  for  the  specific 
design  tested. 

•  Having  a  human  in  the  loop  does  not  mean  that  human  can  necessarily  control  everything 
simultaneously.  Successful  tele-operations  must  allow  for  variable  and  task  dependent  autonomy 
(Stanton). 

•  Alternatives  to  tele-presence  environments  for  remote  control  are  declarative  user  interfaces 
(Trouvain).  Specialists  operate  robots  but  commanders  must  specify  navigational  and  manipulation 
tasks  at  a  higher  and  more  overall  task  objective  oriented  level.  This  may  be  an  advantage  for 
operating  swarms  of  robots  in  which  tele-presence  at  multiple  locations  becomes  impossible. 
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Vehicle  simulation: 

•  An  interesting  application  of  VR  in  the  field  of  vehicle  simulation  is  assembly  and  maintenance 
training.  These  applications  require  intuitive  manipulation  techniques  for  object  handling,  such  as 
virtual  hand  control.  The  state  of  the  art  of  existing  interface  technologies  are  still  far  from  meeting 
functional  requirements  (Vogelmeier).  We  still  have  to  work  with  data  gloves,  head-mounted  displays 
inaccurate  tracking  devices  and  rudimentary  force  feedback  systems.  No  interaction  method  can 
generally  be  applied.  Given  the  limitations  of  current  technology  we  have  to  fall  back  to  task  and 
subtask  specific  interface  designs.  For  many  purposes,  real  mockups  are  still  the  best  way  to  go. 
There  is  an  urgent  need  to  explore  new  ways  of  interfacing  with  virtual  worlds,  for  example  brain 
machine  interaction. 

•  An  important  aspect  of  vehicle  simulation  is  motion  cuing,  the  simulation  of  forces  that  a  vehicle 
exerts  on  the  user.  In  particular,  for  virtual  training  motion,  cuing  may  be  of  crucial  importance  for 
training  transfer.  Evidence  for  this,  however,  is  sparse.  A  very  advanced  experimental  motion 
platform  (Desdemona)  with  all  degrees  of  freedom  and  the  option  of  sustained  3G  (max)  is  currently 
being  built  at  TNO  in  the  Netherlands  (de  Graaf).  It  will  be  used  for  research  regarding  the 
characteristics  of  human  motion  senses  and  for  studying  the  added  value  of  motion  cuing  for  various 
driving  and  flight  simulations.  For  the  study  of  joint  mission  training  requirements  it  has  been  linked 
to  real  F16  cockpits,  C2  facilities  and  tele-presence  control  systems. 

•  Fully  virtual  training  environments  seem  to  no  longer  meet  the  functional  requirements  for  vehicle 
simulation.  Furthermore,  head  mounted  displays  have  been  shown  to  destroy  communication  between 
crew  members.  Consequently  a  mix  of  live,  virtual  &  constructive  environments  (LVC)  are  being 
explored  for  the  evaluation  of  future  armored  vehicles  (Espenant).  LVC  enables  rapid  prototyping  of 
vehicles  and  can  be  used  to  assess  future  technologies  that  are  currently  planned.  Unfortunately, 
objective  performance  measures  to  determine  the  effectiveness  of  these  new  simulation  concepts  are 
still  lacking. 

Dismounted  soldier  simulation: 

•  The  dynamics  of  real  missions  has  not  yet  been  sufficiently  captured  in  dismounted  virtual  simulators. 
Although  technology  is  not  rigorous  enough  for  effective  simulations  of  many  dismounted  tasks,  the 
main  causes  of  limited  support  for  the  development  and  adoption  of  training  simulators  for 
dismounted  soldiers  are  cultural  (Jones).  Leaders  want  to  train  in  real  environments,  soldiers  want  to 
“go  out  and  get  cold”.  It  is  simply  not  the  same  in  a  simulator.  Further  research  should  reveal  to  what 
extent  exciting  scenarios,  new  forms  of  role  playing  and  the  “x-factors”  from  the  gaming  industry  can 
attract  game  savvy  soldiers  into  virtual  simulators  Experiments  made  clear  that  game  playing 
experience  has  implications  for  the  motivation  of  trainees  and  the  effectiveness  of  training  (Knerr). 
The  next  generation  trainees  are  gamers  who  don’t  like  games  that  have  been  designed  for  training 
(not  enough  fun).  Currently  a  big  gap  exists  between  gaming  and  training  applications.  If  tomorrow’s 
military  are  experienced  gamers,  will  training  solutions  that  use  game  technology  be  sufficiently 
engaging  for  training  purposes? 

•  Formerly,  human  factors  experts  thought  they  knew  better  what  was  good  for  the  user  than  the  user 
did.  Nowadays  users  are  heavily  involved  in  all  phases  of  the  evolutionary  development  process  of 
successful  training  simulators.  It  is  important  to  let  training  systems  adapt  organically  and  to  evaluate 
on  the  fly  (Sadagic).  The  combination  of  many  users  and  short  design  cycles  provides  the  feedback 
necessary  to  design  and  develop  effective  training  simulators. 

•  We  form  cohesive  teams  of  soldiers  by  letting  them  share  experiences  and  stories.  VR  is  a  potentially 
powerful  training  tool  to  support  these  processes  (Brooks).  An  important  question,  however,  is  how  to 


RTO-MP-HFM-136 


T  -  9 


Technical  Evaluation  Report 


ORGANIZATION 


built  cost-effective  training  systems?  An  approach  proposed  by  Brooks  is  to  first  built  effective 
systems  and  than  cost  reduce,  not  the  other  way  around.  The  challenges  for  the  coming  years  are  to 
bring  team  members  in  one  space.  How  to  provide  personal  displays  without  head-Mounted 
Displays?  How  to  track  each  of  the  members  (diagnostics)  and  how  to  built  in  “pinball  scores”  to 
make  the  training  attractive  are  questions  that  need  to  be  answered.  A  success  factor  is  to  involve 
users  from  the  start  to  ensure  successful  early  adoption  of  the  training  system.  Technological 
challenges  are  rapid  and  cheap  scenario  generation,  model  acquisition  by  laser  and  video  and  scenario 
capture  by  computer  vision. 

•  Dismounted  soldier  simulators  are  in  urgent  need  of  intuitive  interfaces  for  navigating  the  virtual 
world.  A  good  example  of  human  factors  engineering  has  led  to  an  innovative  Sony  game  console 
based  navigation  interface  (Templeman)  that  matches  the  natural  motion  of  soldiers  in  the  field.  High- 
fidelity  navigation,  however,  remains  an  issue  and  requires  the  combination  of  real  movements  in 
virtual  environments  which  needs  further  research. 

•  Mission  rehearsal  is  a  subset  of  training  and  may  require  less  fidelity  than  training.  Simulation  has 
also  been  set  up  for  pilot  selection  and  is  considered  a  part  of  tests  for  promotion.  In  general  the 
emphasis  on  dismounted  soldier  training  is  on  cognitive  skills.  For  this  purpose,  low-end  simulations 
and  game  use  are  on  the  rise. 

Emerging  applications  fields: 

•  VR  for  selection  of  personnel  (suggestion  Milgram). 

•  VR  for  medical  assessment,  cognitive  tests,  and  treatment  (Pair). 

•  Gaming  environments  for  recruitment  (US  Army  game). 


Requirements  perspective 

•  The  US  Army  TRADOC  Program  Integration  Office,  Virtual  (TPIO, Virtual)  (Shufelt)  has  developed  a 
clear  vision  and  strategy  on  future  combat  training  systems.  Key  for  future  virtual  training  systems  will  be 
that  they  are:  combined  arms,  full  spectrum,  embedded  and  all  terrain.  In  2025  approximately  10%  of  the 
total  force  should  be  able  to  train  according  to  these  principles.  We  will  see  fading  boundaries  between 
training  and  operations.  Embedded  training  devices  will  also  serve  to  realize  operational  networked 
enabled  capabilities  for  net  centric  warfare.  The  most  important  technology  trends  identified  are  the  rise 
of  joint  experimentation  networks,  the  use  of  (commercial)  game  technology,  the  development  of  virtual 
humans  and  augmented  reality  displays.  Requirements  for  these  developments  are  interoperability,  “plug 
and  train”  functionality,  the  availability  of  terrain  databases  within  48  hours,  multilevel  security  and 
cognitive  models  for  virtual  instructors. 

•  The  US  Army’s  PEO,STRI  (Riggins)  wants  to  strengthen  expeditionary  capabilities  by  developing 
trainings  systems  with  a  highly  modular  composition.  The  strengths  of  current  training  systems  are 
mobility  and  tactical  and  spatial  fidelity.  Areas  for  improvement  are  virtual  instructor  functionality,  train 
as  you  fight  functionality  (combinations  of  live  and  constructive)  and  mission  rehearsal  functionality. 
Progress  is  not  solely  dependent  on  technology  development.  It  has  been  recognized  that  all  lines  of 
development  should  progress  in  balance:  doctrine,  materiel,  leadership,  facilities,  operations  and 
personnel.  The  focus  will  be  on  collective  (not  just  individual)  training  functionality. 
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Application  development  approach 

•  Much  of  the  conclusions  of  human  factors  research  on  the  effectiveness  of  VR  and  mixed  reality  are 
still  in  terms  of  “don’t  know”  and  “depends  on”.  This  illustrates  the  complexity  of  the  research 
questions.  Human  factors  knowledge  traditionally  focused  more  on  individual  tasks  in  controlled 
conditions.  There  are  insufficient  theoretical  frameworks  and  predictive  models  for  more  complex 
socio-technical  systems  and  a  systems  of  systems  approach.  For  training  situations  that  deal  with 
collective  performance  most  human  factors  questions  are  still  unanswered  because  cognitive  models 
of  group  behavior  are  lacking  (Sadagic  and  Darken). 

•  In  order  to  progress  on  research  questions  more  explorative  concept  development  and 
experimentation  methods  are  being  developed.  It  is  a  challenge  to  find  new  methods  for  structuring 
concept  development  and  experimentation  such  that  existing  human  factors  knowledge  is  brought 
optimally  into  the  design  and  such  that  we  do  not  arrive  at  suboptimal  solutions. 

2.2  Bottlenecks  and  opportunities 

Based  on  the  observations  listed  in  section  2.1  we  summarize  the  following  opportunities: 

•  Agendas  and  strategies  of  defense  organizations  world  wide  reveal  a  high  priority  for  the  realization 
of  embedded  virtual  training  capabilities  to  strengthen  the  ability  to  train  while  deployed  to  locations 
around  the  world.  The  following  key  technology  trends  have  been  identified:  joint  experimentation 
networks,  the  use  of  (commercial)  game  technology,  the  development  of  virtual  humans  and 
augmented  reality  displays.  Embedded  training  devices  will  also  serve  to  realize  operational 
networked  enabled  capabilities  for  net  centric  warfare.  The  focus  will  be  on  collective  (not  just 
individual)  training  functionality. 

•  Emerging  applications  fields  are  VR  for  selection  of  personnel,  VR  for  medical  assessment,  cognitive 
tests,  and  treatment  and  gaming  environments  for  recruitment. 

Section  2.1  also  showed  the  following  bottlenecks  and  challenges: 

•  Mixed  reality  (merging  of  real  and  virtual  worlds)  is  promising  but  the  technology  for  tracking  body 
positions  in  the  open  field  and  real  time  real  world  modeling  are  still  immature. 

•  Validated  test  batteries  are  needed  to  assess  cognitive  performance  in  mixed  reality  (attention,  spatial 
ability,  memory,  reasoning  abilities). 

•  Interface  technology  has  not  made  much  progress  in  the  last  six  years  (head-mounted  displays  are  still 
cumbersome  and  destroy  communication,  force  feedback  is  still  in  its  infant  years).  There  is  a  need 
for  improved  ergonomic  design  of  mobile  AR  applications  and  hands  free,  intuitive,  multi-modal 
interfaces  allowing  “train  as  you  fight”  simulations. 

•  For  training  situations  that  deal  with  large  scale  collective  performance  in  networked  operations  most 
human  factors  questions  are  still  unanswered  because  predictive  cognitive  models  of  group  behavior 
are  lacking.  Progress  urgently  requires  the  involvement  of  social  disciplines  (dimensions  of  socio- 
technical  systems,  cultural  factors),  organizational  disciplines  (mechanisms  and  mathematics  of  self 
organizing  networks)  and  creative  disciplines  (scenario  and  gaming  elements). 

•  The  main  causes  of  limited  use  of  the  potential  of  training  simulators  for  dismounted  soldiers  are 
cultural:  leaders  want  to  train  in  real  environments  and  soldiers  want  to  “go  out  and  get  cold”.  The 
case  for  training  in  virtual  worlds  to  prepare  for  training  in  live  environments  has  yet  to  be  effectively 
made. 
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•  Current  training  systems  usually  meet  the  immediate  training  needs  but  are  not  sufficiently  designed 
to  allow  for  flexibility  and  evolution. 

•  The  development  of  many  virtual  training  applications  has  not  yet  balanced  all  lines  of  development: 
doctrine,  materiel,  leadership,  facilities,  operations  and  personnel. 

•  More  interchange  of  ideas  is  needed  between  research  community,  system  developers,  requirement 
people  and  military  users,  especially  in  C2/NEC.  Often  there  is  good  evidence  &  data  to  support  a 
given  decision  but  requirement  people  supporting  another  decision  may  not  be  aware  of  it. 

•  Services  need  to  communicate  better  resulting  in  effective  transfer  of  knowledge,  lessons  learned,  etc. 

•  There  are  no  funds  for  longitudinal  studies.  Longitudinal  studies  are  essential  for  measuring  long 
term  adaptation  of  users  to  new  technology. 

•  Human  factors  conclusions  are  often  design  specific.  There  is  a  strong  need  for  more  generic  guide 
lines  to  guide  developers  and  optimize  system  design. 

2.3  Recommended  actions 

We  have  come  to  the  following  recommendations: 

•  Create  fully  integrated  joint  Concept  Development  and  Experimentation  facilities  for  the  development 
and  training  of  Networked  Enabled  Capabilities  (dynamic  reallocation  of  functions,  switching 
micro/macro  command  levels,  trust  in  command  at  distance,  etc.). 

•  Focus  research  on: 

•  Theoretical  framework  on  socio-technical  systems  and  predictive  cognitive  models  for  the 
effective  structuring  and  evaluating  of  Concept  Development  and  Experimentation  (CDE) 
processes  with  a  focus  on  collective  performance, 

•  Virtual  characters  and  virtual  instructors, 

•  Non-obtrusive  intuitive  multimodal  (brain  machine)  interfaces  for  hands  free  navigation  and 
manipulation,  allowing  (non-verbal)  team  interactions  and  “train  like  you  fight”, 

•  Rapid  scenario  generation  and  scenario  capture, 

•  Non  obtrusive  mixed  (augmented)  reality  displays, 

•  Tracking  technology  for  the  open  field, 

•  Validated  test  batteries  to  assess  cognitive  performance  in  mixed  reality  (attention,  spatial  ability, 
memory,  reasoning  abilities), 

•  Generic  human  factors  guidelines  versus  design  specific  results. 

•  Create  budgets  for  longitudinal  studies  on  adaptation  of  users  to  new  technology, 

•  Involve  organizational,  social  and  cultural  disciplines, 

•  Stimulate  dual  use  of  commercial  gaming  technology  and  military  simulation  technology. 

•  Design  training  systems  to  allow  for  flexibility  and  evolution. 

•  Balanced  all  lines  of  system  development:  doctrine,  materiel,  leadership,  facilities,  operations  and 
personnel. 
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•  Better  organize  the  dialog  between  the  research  communities,  system  developers,  requirement  people 
and  military  users,  especially  in  the  field  of  C2  and  NEC. 

•  Better  organize  the  dialog  between  services  resulting  in  a  effective  transfer  of  knowledge  and  lessons 
learned. 

The  enthusiasm  of  the  workshop  attendees  and  the  evident  willingness  to  share  ideas  and  to  discuss  their 
findings  provides  a  promising  base  for  working  on  these  recommendations. 

Please  note  that  recommendations  below  generated  by  NATO  study  group  HFM-021  in  2000  are  still 
unfulfilled  needs  today: 

•  The  military  should  develop  a  vision  on  the  use  of  VR  technology  and  more  clearly  specify  their 
needs. 

•  Industry  should  work  on  standardization  and  should  substantially  bring  human  factors  into  their 
design  and  development  processes. 

•  Academia  and  research  institutes  should  coordinate  and  accelerate  their  long-term  research  efforts  to 
focus  on  natural  interfaces  (innovative  metaphors)  and  on  how  to  model  human  and  object  behavior. 

•  In  the  short  term  academia  should  focus  on  human  factors  metrics  and  metrics  for  team  and  collective 
performance  (cognition,  communication),  and  a  standard  evaluation  methodology. 

•  In  general,  better  coordination  between  military  organizations,  industry  and  academia  is  necessary  in 
order  to  identify  gaps  in  current  knowledge  and  coordinate  research. 


3.0  CONCLUSIONS 

The  workshop  “Virtual  media  for  Military  Applications”  uniquely  has  brought  together  prominent  people 
from  academia,  industry  and  the  military.  Workshop  attendees  enthusiastically  and  knowledgeably: 

•  exchanged  operational  requirements  and  on-going  research  in  the  field  of  augmented,  mixed  and 
virtual  reality  technology  (AMVE)  for  military  applications, 

•  identified  success  factors  and  bottlenecks  for  implementation,  and 

•  have  made  recommendations  for  future  research  agenda’s,  methodological  approaches  and 
organizational  issues. 

The  main  conclusions  are: 

•  For  training  situations  that  deal  with  collective  performance  most  human  factors  questions  are  still 
unanswered  because  predictive  cognitive  models  of  group  behavior  are  lacking.  Serious  research 
budgets  for  longitudinal  studies,  fully  integrated  concept  development  and  experimentation  facilities 
and  the  involvement  of  social,  organizational  and  creative  disciplines  are  necessary  to  find  the 
answers. 

•  Developers  of  virtual  media  applications  make  significant  use  of  public  data  bases  and  commercial 
off  the  shelf  gaming  technology.  Gaming  technology  evolves  quickly  with  extremely  short  cycles  and 
massive  numbers  of  users.  As  such,  it  generates  implicit  human  factors  knowledge  together  with 
technological  advances  which  seems  to  put  military  training  system  developers  and  the  traditional 
human  factors  community  at  the  “tail”  of  the  gaming  community.  Dual  use  of  gaming  and  military 
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simulation  technology  should  be  further  stimulated.  Military  researchers  should  not  try  to  keep  pace 
with  gaming  technology  but  rather  work  to  adapt  technology  advances  to  unique  military  applications. 

•  Exciting  pushes  are  seen  into  “new”  territories  such  as  serious  gaming,  unmanned  vehicles  and 
robotics  (tele-presence),  dismounted  training  capabilities  and  emerging  applications  in  the  field  of 
personnel  selection  and  recruitment  and  medical  treatment. 

•  The  dialog  between  the  research  communities,  system  developers,  requirement  people  and  military 
users  and  between  services  themselves  has  to  be  improved  for  a  more  effective  transfer  of  knowledge 
and  lessons  learned. 


Future  Meetings 

In  2000,  HFM  121  held  a  workshop  to  look  at  the  applications  of  virtual  technologies  to  military  use.  In  many 
respects  HFM-136  was  a  re-look  at  an  expanded  realm  in  which  AMVE  technologies  are  being  used  by  the 
military.  Six  years  have  gone  by  but  as  noted  above  much  work  still  needs  to  be  done  to  provide  NATO 
militaries  with  AMVE  systems  that  are  usable  by  soldiers,  sailors  and  airman  and  effective  for  the  purpose 
they  were  designed. 

The  direction  of  change  has  taken  some  unanticipated  turns  since  2000.  Full  scale  VR  does  not  seem  as  likely 
a  solution  as  once  thought.  Game  technology,  augmented  reality,  and  embedded  capability  seem  to  be  the 
direction  requirements  and  technology  is  going.  The  need  to  ensure  that  human  considerations  are  taken  into 
account  is  a  continuing  requirement.  There  will  need  to  be  a  follow  on  to  this  workshop  that  explores  in  more 
detail  the  current  trends  as  noted  in  this  TER.  In  particular  the  embedding  of  training  in  weapon  systems  is  a 
topic  that  will  include  considerations  of  live,  virtual  and  constructive  simulation  and  will  be  worthy  of  further 
study. 
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